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■^j- ' Abstract In this paper, a class of coupled systems of nonlinear Schrodinger equations 
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with sign-changing potential, including the linearly coupled case, is considered. The exis- 
Oh ' tence of non-trivial bound state solutions via linking methods for cones in Banach spaces 

< 

is proved. 
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1 Introduction and mam results 

O 

00 

Recently, many mathematicians focused their attention to coupled nonlinear Schrodinger 

in ■ 

systems. From the viewpoint of physics, coupled Schrodinger systems arise from the 
models of a lot of natural phenomena. A typical example is the study of the dynamics of 
coupled Bose-Einstein condensates and the following equation is derived 

J i§ W = (~d 2 /dx 2 + V X + UnHi? + U^ 2 )^ + X^ 2 , 

5-H 



i = (~d 2 /dx 2 + v 2 + u 22 \H 2 + u 2 i\ipi\ 2 )ip 2 + Mi- 



(i.i) 

— ( i&^i _i_ t/, _i_ rr__u/._l^ _i_ rr„ la. _l \.u. 

at 

Such systems of equations also appear in nonlinear optical models and many other physical 
contexts, see [7] for detail discussions. For such coupled systems, the solutions of the form 
ipj = Uj exp(iujt) (standing waves) are interesting, where Uj solve the following system 

■^ + (V 1 +uj 1 )u 1 = -(U u \u 1 \ 2 + U 12 \u 2 \ 2 )u 1 -Xu 2 , 
2 (1.2) 

+ (V 2 + uj 2 )u 2 = -(U 22 \u 2 \ 2 + U 21 \ Ul \ 2 )u 2 - Xu v 
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In this paper, we will consider the following coupled system of nonlinear Schrodinger 
equations 

— Aui + (pi(x) - XVi(x))ui = Wt{x,ui,u 2 ) + Xj(x)u 2 , 
< -Au2 + (b2(x)-XV 2 (x))u2 = W s (x,u 1 ,U2) + X^(x)u 1 , (1.3) 
^ Ui,«2 G il 1 (R^), 

here and in the sequel, K G L 00 (R 7V ), 7 G ^(R^), * = 1,2, = (W t ,W 8 ) is the 

gradient of W(x, t, s) with respect to z = (t, s) G R 2 and we will write W(x, z) = t, s) 
for convenience. We divide our discussions into two cases. 

The non-radially symmetric case. We assume 6j(x) satisfying the following conditions 

(B) for i = 1,2, hi G C(R N ), there exists a constant fe? > such that inf bi(x) > 6°, 
and the n dimensional Lebesgue measure meas{x G R" | 6j(x) < M} < 00 for any 
M > 0. 

We assume VF satisfying the following conditions. 

(Wi) W G C^R^ x R 2 ), there exists p G (2,2*) such that < W{x,z) < (7(1 + |z| p ), 
V (x, 2) G H N x R 2 , here, 2* = ^ if JV > 2 and 2* = +00 if N = 1, 2, 
W(x,z) 

(W2) lim — — 7^ — = +00 uniformly for x G R^, 

|z|-Kx> \z\ z 

W(x z) 

(W 3 ) W t (x, 0, s) = 0, W fl (x, t, 0) = for any x G R^, s G R, t G R, and lim , v ' ; = 

|.z|->0 \Z\ Z 

uniformly for x G R^, 

(W 4 ) set W(x,z) = V z W(x,z)-z-2W(x,z), then there exists 9 > 1 such that 6>W(x,z) > 
W(x, r]z) , V (x, z) G R^ x R 2 and 77 G [0, 1] . 

Remark. (1) From (W4) and W(x,0) = 0, we see that W(x,z) > for any (x,z) G 

x R 2 by taking rj = 0. So we have V 2 lf(x, z) ■ z > 2W(x, z). 
(2) From condition (W3), when X r y(x) 7^ 0, Vx G R^, for a non-trivial solution u = (m, 1*2) 

of the problem (|1.3p . it is easy to see that u\ 7^ and U2 / 0, so u does not have 

an immediate counterpart for a single equation. We also remind that under the above 

conditions the potential bi(x) — AV^(x) may change sign since A G R, see Theorem 11.11 

below. 

In this case, we have the following main result. 

Theorem 1.1 If (B) and (Wi)-(W4) hold, the problem HI. 3$ possesses a non-trivial so- 
lution for every X G R. 
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The radially symmetric case. We assume that &i(x) satisfy the following condition 

(B) r for £ = 1,2, bi E C(R N ), there exists a constant W > such that inf bi(x) > b®, 

x€R N 

and bi are radially symmetric, i.e., bi(x) = 6j(|x|),Vx G R^, 
and Vi(x), 7 (x), W(x,z) further satisfy 
(V) r for i = 1,2, = Vi(\x\), 7 (x) = 7 (|x|),Vx G R N . 

(W 5 ) W(a;,z) = iy(|x|,z), V(x,z) G R w x R 2 . 
For this case we have the following result. 

Theorem 1.2 // (B) r , (V) r and (Wi)-(Ws) hold, the problem (j 1.3§ possesses a non- 
trivial radially symmetric solution for every A G R. 

Next, we consider some special cases of (jl.3p . Firstly, we consider some linearly coupled 
systems. Precisely, we assume that Wt(x,t,s) dose not depend on s and W s (x,t,s) does 
not depend on t, that is to say one can write (|1.3p as 

-Aui + (h(x) - \Vi(x))ui = f(x,u\) + A 7 (x)u 2 , 
< -Au 2 + (b 2 (x) - XV 2 (x))u 2 = g(x, u 2 ) + A 7 (x)ui, (1.4) 
k «1,«2 G ^(R^). 

In this case, we assume that /, g G C^R^ x R) satisfy 

(ft) 3p! G (2,2*) such that |/(x,i)| < C(l + |i| Pl_1 ), f(x, t)t > 0, V (x, t) G H N x R, 

(f 2 ) set F(x,t) = f n * f(x,t)dt, lim — [ ' - = +oo uniformly in x G R^, 

u |t|->oo |tp 

(fa) lim ^ ' - = uniformly in x G R^, 

(fi) .F(x, t) = /(x, i)i — 2F(x, t), then there exists #i > 1 such that 9\J-{x, t) > J-(x, rft), 
V(x,t) GR W xRandr/G [0,1], 

(gi) G (2,2*) such that \g(x,s)\ < C(l + |s| P2_1 ), 5(x,s)s > 0, V(x,s) G R^ x R, 

a G(x,s) 
(g 2 ) set G(x,s) = [1? g(x, s)ds, lim — — ^ — = +oo uniformly in x G R^, 

u |s|->oo \s\ z 

(g3) lim - — = uniformly in x G H. N , 

s-s>0 s 

(g 4 ) Q(x, s) = g(x, s)s — 2G(x, s), then there exists 9 2 > I such that 9 2 Q(x,s) > Q(x,r)s), 
V (x, s) G R^ x R and rj G [0, 1]. 

3 



Theorem 1.3 If (B), (fi)-(f4) and (gi)-(g4) hold, the problem (\1.4\ ) possesses a non- 
trivial solution for every A G Ft. 



Proof . Set W(x, t, s) = F(x, t) + G(x, s), it is easy to see that (Wi) and (W4) hold. 

1*1 



As for (W 2 ), from (f 2 ) and (g 2 ), VM > 0, there exists R > such that ^fe^ > 2M 



when \t\ > R and > 2M when |s| > R. Then 



F(x,t) + G(x,s) F(x,t) + G(x,s) 
t 2 + s 2 - 2max(|t| 2 ,|s| 2 ) 

W(x, z) 

when max(|i|, Isl) > R. So lim — — ^ — = +00 uniformly for x G R^. 

|z|^oo |z| 2 

From (£3), (g3) and the continuity of / and g, we can see /(x, 0) = = g(x, 0), so 

W t (x, 0, s) = 0, W s (x, i, 0) = for any x G R^, s G R, t G R. Also from (f 3 ) and (g 3 ), we 

have lim — f X ' — = and lim — r ,' - = 0, so 
|*|—^0 \t\ 2 |s|->0 \s\ 2 

F(x,t) + G(x,s) F(x,t) G(x,s) 
\t\ 2 + \s\ 2 - \t\ 2 |s| 2 

So (W3) holds. From Theorem ll.il we get the assertion. I 
As in Theorem II .2\ assuming that f(x,t) and g(x,s) further satisfy 

(f 5 ) f(x,t) = f(\x\,t), for any (x,t) G H N x R, 
(gs) g(x,s) = g{\x\,s), for any (x,s) G H N x R, 

also setting W(x, i, s) = F(x, t) + G(x, s) and by the same reason as in the proof of 
Theorem 11.31 we have the following consequence. 

Theorem 1.4 If (B) r , (V) r , (fi)-(fs) and (gi)-{gs) hold, the problem < \l-4\) possesses a 
non-trivial radially symmetric solution for every A G R. 

By taking f(x,t) = ci{x)\t\ Pl - 2 t and g(x,s) = c 2 (x)\s\ P2 - 2 s with a G L 00 ^) and 
inf Ci(x) > 0, i = 1,2, we get the following system 

x£K N 

-Ait! + (61 (x) - AVl(x))ui = ci(x)|ui| Pl " 2 ni + A7(x)u 2 , 
-Au 2 + (6 2 (x) - XV 2 (x))u 2 = c 2 (x)\u 2 \ P2 - 2 u2 + A 7 (x)ui, (1.5) 
u,v G ff^R^), 

then for pi,p 2 G (2, 2*), we have the following consequences. 

Corollary 1.5 // (B) holds, the problem i\1.5\) possesses a non-trivial solution for every 
A G R. 
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Corollary 1.6 // (B) r , (V) r hold, and Ci(x) = for any x G R , i = 1,2, i/te 

problem (|i.5|) possesses a non-trivial radially symmetric solution for every A G R. 

Secondly, by taking W(x, i, s) = \t A + ^t 2 s 2 + \s 4 , we get the following systems 



(1.6) 



— Aui + — AVi(x))ni = u\ + u\u\ + A7(x)«2j 

-Au 2 + (62 (^) - AV 2 (a;)) / U2 = u 2 + ^f^2 + A7(x)«i, 
ui,«2 G fl^R*). 
as consequences of Theorem 11.11 and 11.21 we have 

Corollary 1.7 If (B) holds, the problem \1.6\ possesses a non-trivial solution for every 
A G R. 

Corollary 1.8 If (B) r and (V) r /10W, the problem (| i.6'|) possesses a non-trivial radially 
symmetric solution for every A G R. 

The study of linearly coupled Schrodinger systems from the mathematical point of 
view began very recently, see [U HI [7]. In [3], the authors proved the existence of 
positive ground state solution of the following system of nonlinear Schrodinger equations 
for < A < 1, 

-An + u = (l + a(x))\u\ p ~ 2 u + Xv, 

-Av + v = {l + b(x))\v\P- 2 v + Xu, (1.7) 
u,v G H l (R N ), 

with a,b G ^(R^), lim a(z) = lim 6(x) = 0, inf{l + a(x)} > 0, inf{l + b(x)} > 

|z|->oo |x|-Kx> R v R v 

and a(x) + 6(x) > 0. In [3J, the authors devoted to the study the multi-bump solitons of 



the following system 



—Ait + u — u 3 = ev , 
— Av + v — v 3 = eu, 



(li 



in R^ with dimension N = 1, 2, 3. In [T], A. Ambrosetti studied the following two systems 



-li'i + Hi = (1 + ea\(x))u\ + 7U2, 
—u'2 + u 2 = {1 + ea 2 (x))u 2 + 7«i, 
ui,ti 2 G tf^R), 

-e 2 ^' + u\ + f7i(a;)ni = uf + 7u 2 , 
-e 2 u 2 +u 2 + U 2 (x)u 2 = u 2 + 7«i, 
iti,it 2 G F X (R), 



(1.9) 



J1.10) 



and proved the existence of non-trivial solution for (jl.9p under the conditions a% G L°° (R) , 
lim aj(x) = 0, i = 1, 2, < 7 < 1, 7 / 3/5, and (jl.lOp possesses a solution concentrating 
at nondegenerate stationary points of the sum U\ + U2 when e — > under the conditions 
Ui G and inf Ui(x) > —1, i = 1,2. The main tools in [TJ El E] are the perturbation 
techniques, we refer [5] for ditailed discussions about these methods. In [TJ, the following 
system was considered 

— + a(x)ui — b(x)ii2 = c{x)H\(v,\,U2)ii\, 
< — ti' 2 ' + d(x)ii2 - e(x)ui = f(x)H2(ui,U2)u 2 , 0-H) 
ui,u 2 G H 1 (R), 

the authors got a non-trivial solution via Krasnoselskii fixed point theory. We note that 

the potentials in systems (|1.7p - (|l.ll|) are positive. 

To prove the main theorem, we deal with the existence problem of non-trivial solutions 

by variational methods. We first study an eigenvalue problem, whose eigenfunctions are 
solutions of (jl.3p but without the nonlinear term, then the non-zero critical point of 
the functional related to the nonlinear perturbation of this eigenvalue problem is a weak 
solution of (ll.3j) . To find the critical point, we use a critical point theorem developed by 
Degiovanni and Lancelotti in |10j . 

The rest of the paper is organized as follows. The variational setting is contained in 
section 2. In section 3, we study the eigenvalue problem. We prove that there exists a 
divergent sequence of eigenvalues which are defined by the cohomological index. We prove 
Theorem 11.11 and 11.21 in section 4. 

2 Variational setting 

Let Hi := {u\ G i? 1 (R Ar )| J RJV bi(x)ufdx < 00}, then Hi is a Hilbert Space with inner 

product (iii,i>i)i = J R jv(Vn! -Vvi + bi(x)uiVi)dx and norm [|iti||j = Similarly, 

let H2 := {U2 G H 1 (R N ) \ J-dn 62(^)^2 ^ x < °°}> then H2 is a Hilbert Space with inner 

product (u2,v 2 )2 = J R iv(Vu 2 • Vv 2 + ^(^O^^dx and norm ||it 2 1|| = (^2,^2)2- 

For the non-radially symmetric case, by the condition (B), Hi and H2 can be compactly 

embedded into L P (R N ), 2 < p < 2* (see for example, 0E7J). Set H:= H x x H 2 , then U is 

a Hilbert space with inner product (•, •) = (•, + -)2 and with norm ||u|| 2 = ||ui||i+||«2||l 

for u = (ni,n 2 ). 

For the radially symmetric case, let Hi^ r := {ui G Hi\ui is radially symmetric}, 
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H 2r := { u 2 G -^21^2 is radially symmetric}, then Hi^ r is a Hilbert Space with inner 

product (•,•)« and norm || • ||j for i = 1,2. By condition (B) r , Hi^ r can be compactly 

embedded into L P (R N ), 2 < p < 2* for i = 1,2 (see [SI [IT]). In this case, we set 

H r := H\ r x -f^.n then T~L r is a Hilbert space with inner product (•, •) = (•, -)i + (•, -) 2 and 

with norm ||u|| 2 = ||ui|| 2 + 1 1 1 1 1 f° r u = { u \i u 2)- 

In order to prove Theorem I Lit we define a functional ^ : H — > R by 

(u) = E(u) - AJ(u) - P(u), u = (u x , u 2 ) G (2.12) 

where 

£(u) = i||u|| 2 , (2.13) 

J(u) = J + 7 (x) Ul n 2 + ^ 2 (x)n^ ds, (2.14) 

and 

P(u)= I W(x,u)dx= I W(x, Ul ,u 2 )dx, (2.15) 
then these four functionals are C 1 , and for u = (u\,u 2 ), v = (^1,^2) 6 "H, there hold 

(P/(u),v)=/ (Vu 1 -X7v 1 + b 1 (x)u 1 v 1 )dx+ (Vu 2 -Vv 2 + b 2 (x)u 2 v 2 ) dx, (2.16) 

(J'(u),v) = / (Vx(x)uivi + j(x)u 2 vi +^(x)u\v 2 + V 2 (x)u 2 v 2 )dx, ( 2 -l?) 

(P'(u),v)=/ (Wi(x,«i,«2)vi + W«(x,«i,«2)t^)<Lr, (2.18) 

<*'(u),v) = <P'(u),v) - A(J'(u),v) - <P'(u),v). (2.19) 

It is clear that critical points of \l/ are weak solutions of (|1.3p . 

For the radially symmetric case, we can also define these four functionals and (|2.16p - 

()2. 19|) hold, the only difference is the domain T~L of the functional ^ is replaced by % r . 

And the critical points of the functional ^> are radially symmetric weak solutions of f| 1 . 3 1) . 
In order to find a critical point of ^, we need the following critical point theorem. It was 

proved in [TO], where the functional was supposed to satisfy the (PS) condition. Recently, 

in [S], the author extended it to more general case (the functional space is completely 

regular topological space or metric space). As observed in |15| . if the functional space is 

a real Banach space, according to the proof of Theorem 6.10 in [9], the Cerami condition 

is sufficient for the compactness of the set of critical points at a fixed level and the first 

deformation lemma to hold (see [16J). So this critical point theorem still hold under the 

Cerami condition. 
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Theorem 2.1 (110]) Let % be a real Banach space and let C_ ; C+ be two symmetric 
cones in % such that C+ is closed in %, C_ Pi C+ = {0} and 

i(C_ \ {0}) = i(H \ C+) = m < oo. 

Define the following four sets by 

D_ = {u£ C_| ||u]| < r_}, 
S + = {u 6 C+| ||u|| = r + }, 

Q = { u + te\u £ C-,t > 0,\\u + te\\ < r_}, e € % \ C_, 
F = { n + ie| -u G C_, t > 0, ||it + te\\ = r_}. 

T/ien (Q,D_ U -ff) ZinA;s S 1 -^ cohomologically in dimension m + 1 over Z2. Moreover, 
suppose ^ S C 1 (^,R) satisfying the Cerami condition, and sup ^(x) < inf ^(x), 
sup^I / (x) < 00. Then ^ has a critical value d > inf ^(x). 

x&Q a;65+ 

For convenience, let us recall the definition and some properties of the cohomological 

index of Fadell-Rabinowitz for a Z2-set, see [1 1 |, 1121 [T6] for details. For simplicity, we only 

consider the usual Z2-action on a linear space, i.e., Z2 = {1,-1} and the action is the 

usual multiplication. In this case, the Z2-set A is a symmetric set with — A = A. 

Let E be a normed linear space. We denote by S{E) the set of all symmetric subsets 

of E which do not contain the origin of E. For A G S{E), denote A = A/7j2- Let 

p : A ->• RP°° be the classifying map and p* : H*(RP°°) = Z 2 [w] -)■ H*(A) the induced 

homomorphism of the cohomology rings. The cohomological index of A, denoted by i(A), 

is defined by sup{A; > 1 : p*(uj k ~ 1 ) 7^ 0}. We list some properties of the cohomological 

index here for further use in this paper. Let A, B G S(E), there hold 

(11) (monotonicity) if h : A — > B is an odd map, then i(A) < i(B), 

(12) (continuity) if C is a closed symmetric subset of A, then there exists a closed 
symmetric neighborhood N of C in A, such that i(N) = i(C), hence the interior of 
N in A is also a neighborhood of C in A and i(intiV) = i(C), 

(13) (neighborhood of zero) if V is bounded closed symmetric neighborhood of the 
origin in E, then i(dV) = &i~mE. 
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3 The eigenvalue problem 

First we solve the eigenvalue problem 

E'(u) = fif(u), u€n. 



(3.20) 



Lemma 3.1 For any u = (ui,u 2 ), v = (i>i,i>2) G rl, it holds that 

(E'(u) - E'(v),u- v) > (KHi - \\ Vl \\l) 2 + (||n 2 || 2 - |M| 2 ) 2 . (3.21) 
Proof . By direct computations, we have 
(E'(u) -E'(v),u-v) 

= J RN (|Vm| 2 + |Vvi| 2 - 2Vui • Vvx) dx + J RN h(x) (\ Ul \ 2 + \ Vl \ 2 - 2u 1 v 1 ) dx 
+ /rat (l Vn 2| 2 + \Vv 2 \ 2 - 2Vu 2 ■ Vv 2 ) dx + J RN b 2 (x) (\u 2 \ 2 + \v 2 \ 2 - 2u 2 v 2 ) dx. 

From the definition of the norm in Hi, we can get 

f RN (|Vui| 2 + |Vui| 2 - 2Vui • Vvx) dx + f RN bi(x) (|iii| 2 + M 2 - 2«i«i) dx 

= ||m|| 2 + |M| 2 - 2(m, Wl ) a > |M| 2 + \\ Vl \\ 2 - 2||u 1 || 1 |M| 1 = (IKIl! - H^llx) 2 , 

(3.22) 

J r n (|Vn 2 | 2 + |Vv 2 | 2 - 2Vu 2 • Vv 2 ) dx + J RJV b 2 (x) (\u 2 \ 2 + |u 2 | 2 - 2u 2 v 2 ) dx 

= IKIII + IH|| ~ 2(m 2 , v 2 ) 2 > \\u 2 \\ 2 2 + ll^lll - 2||n 2 || 2 ||?j 2 || 2 = (||n 2 || 2 - ||w 2 || 2 ) 2 . 

(3.23) 

Now (^22]) and (1333]) imply fl33T|) . I 
Lemma 3.2 If u n — * u and (i?'(u„),u„ — u) — )• 0, i/ien u n — > u in 

Proof . Since % is a Hilbert space and u n = (u n ,v n ) — 1 u = (u,v), we only need to show 
that ||u ra || —7- ||u||. Note that 

lim (£'(un) " #'(u),u„ - u) = lim ((S'K),^ - u) - (E'{u),u n - u)) = 0. 

n— >oo n— >oo 

By inequality ()3.2ip we have 

(E'(u n ) - E'{u),u n - u) > (K||i - UnllO 2 + (K|| 2 - |M| 2 ) 2 . 

So ||^n||i - > \\ u \\ii ll^nlb - > \\ v \\2 an d hence ||u ra || — > ||u|| as n — > oo and the assertion 
follows. I 
Lemma 3.3 J' is weak-to- strong continuous, i.e. u n — u in % implies J'(u n ) — > J'(u). 



Proof . Since u n = (u n ,v n ) — ^ u = in 7~L, u n — u in H\. So u n — > u in L 2 (R Af ) 

because H± compactly embedded into L 2 (R N ). Similarly, we have v n — > v in L 2 (R JV ). 
For any v = (u, v) G 

I u 2 dx < -q / 6i(x)tt 2 dx < 7o[|tt[|i < All v l| 2 > 

i i 
so ( J RJV -u 2 dx) 2 < C||v||. Similarly, we have {J-rn v 2 dx) 2 < C||v||. Then, 

|(j'(u n )-J'(u),v)| 

= / ( Vi (z) (it n - ti)u + 7(x) (v n - w)tt + 7(x) (tt n -u)v + V 2 (x) (-u n - dx 

< || Vi ||oo ^ ^{u n - u) 2 dx^ ^y^n 2 dx^ + I^H^ (J ^(v n - u) 2 dx^ fj N ^ 2 dx\ 



i i i 

\2 j„ \ I I .~;2j„ I i iita II I / /„. „.\2j„ \ I I „~,2, 



+II7IU / (u n - -u) dx / dx + HV2II00 / - v) dx / u dx 

'R" / V^R^ / \Jk n J \Jr n 



< Ci^J (u n -n) 2 dxj 2 ||v|| + c(j (v n -v) 2 dxj ||v|| 0, 

hence J'(u n ) — ► J'(u). I 
Lemma 3.4 Z/u n ^ u inV,, then J(u n ) — >• J(u). 

Proof . 

2| J(u„) - J(u)| = |(J'(u n ),u n )-(J / (u),u)| 

= |(J'(u n ) - J'(u), u n ) + ( J'(u), u n - u)| 

< ||J'(u n )- J'(u)||||u n || +0(1). 

Because u n — u, u n is bounded. From Lemma 13.31 we have J(u n ) — > J(u). I 
In this section, we assume that V\ and V2 satisfy the following condition 

(**) meas{x G R^l Fi(x) > 0} > or meas{x G R^l V 2 (x) > 0} > 0. 

Set Ai = {u G "H| «/(u) = 1}, by (**), we can see that Ai is not empty, see also Lemma 
GE2 below. Clearly, J(u) = ~(J'(u) , u) , so 1 is a regular value of the functional J. Hence 
by the implicit theorem, Ai is a C 1 -Finsler manifold. It is complete, symmetric, since J 
is continuous and even. Moreover, is not contained in Ai, so the trivial Z2-action on Ai 
is free. Set E = E\m- 

Lemma 3.5 If u G Ai satisfies E(u) = fx and E'(u) = 0, then (fx, u) is a solution of the 
functional equation \3.20\) . 
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Proof . By Proposition 3.54 in |16j . the norm of E'(u) G T*.M is given by ||£"(u)||* = 
min ||-E'(u) — ^j'(u)||* ( here the norm II • II* is the norm in the fibre T*A4, and II • II* is the 
operator norm, the minimal can be attained was proved in Lemma 3.55 in [TB] ). Hence 
there exists v G R such that E'(u) — vJ'(u) = 0, that is (v, u) is a solution of the equation 
(MP and fi = E(u) = ^{E'(u), u) = \{vJ'(u), u) = %(J'(u), u) = uJ(u) = v. I 

Lemma 3.6 E satisfies the (PS) condition, i.e. if (u k ) is a sequence on Ai such that 
E(u k ) — > c, and E'(u k ) — > ; then up to a subsequence u k — > u G Ai in 7-1. 

Proof . First, from the definition of E, we can deduce that (u k ) is bounded. Then, up 
to a subsequence, u k converges weakly to some u, by Lemma [331 we have J(u) = 1, so 
u£ M. 

From E'(u k ) —> 0, we have E'(u k ) — u k J'(u k ) — > in % for a sequence of real numbers 
(vk)- So (£"(ufe) — u k J'(u k ),u k ) — > 0, thus we get v k — > c. By Lemma [3.3| we have 
E'(u k ) — > cJ'(u). Hence 

(E'(u k ),u k - u) = (E'(u k ) - cJ'(u), u k - u) + (cJ'(u), u k - u) -> 0. 

By Lemma 13.21 we obtain — > u. I 
Let T denote the class of symmetric subsets of At, F n = {M G F\ i(M) > n} and 

/j, n = inf sup E(u). (3.24) 

Since J" n D J" n +l, /^n < 

Lemma 3.7 // (**) holds, then for every T n , there is a compact symmetric set M G T n . 

Proof . We follow the idea of the proof of Theorem 3.2 in [13]. Suppose meas{x G 
R | V\(x) > 0} > 0, it implies that Vn G N, there exist n open balls (Bi)\<i< n in R^ such 
that P>i C\ Bj =$ for i ^ j and meas({x G R^| V\(x) > 0} n -Bi) > 0. Approximating the 
characteristic function Xi of set {x G R^| Vi(x) > 0} n Bi by a C°°-function m in L 2 (R Ar ), 
and require that the sequence {«i}i<i< n C C°°(R-' V ) satisfies J Rjv Iq(x)|tij| 2 dx > for 
all i = 1, •■■ , n and suppu, n supp-Uj = when i ^ j. Set Uj = (ui,0) G H, then 
J(uj) = \ J rN Vi(x)\ui\ 2 dx > 0. Normalizing Uj, we assume that J(uj) = 1. Denote by 

n ra 

J7 n the space spanned by (uj)i<j< n . Vu G U n , we have u = ^ a,Uj and J(u) = Yl \ a i\ 2 - 

i=l t=i 

i 

So (J(u)) 2 defines a norm on [/„. Since f/ n is n dimensional, this norm is equivalent to 
|| • || . Thus {u G U n \ J(u) = 1} C Ai is compact with respect to the norm || • || and by 
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the property (i3) of cohomological index, i({u G U n \ J(u) = 1}) = n. So {u G U n \ J(u) = 

1} G T n . If meas{x G R^l V^x) > 0} > 0, the proof is similar. I 
By Lemma [3,7| we have < +oo, and by condition (B), there holds p n > 0. Further- 
more, by Lemma 13.61 and Proposition 3.52 in [16], we see that \x n is sequence of critical 
values of E and p n — > +oo, as n — > oo . By Lemma 13.51 we get a divergent sequence of 
eigenvalues for problem (|3.20p . So we have the following result. 

Theorem 3.8 Under the condition (**), the problem Ii3. 20\) has an increasing sequence 
eigenvalues p n which are defined by {3. 24\ ) and p n — > +oo, as n — > oo . 

Lemma 3.9 Under the condition (**), Set 

p n = inf supE'(u), (3.25) 
where = {K G F n \ K is compact}, we have p n = p n . 

Proof . From Lemma [3771 J 7 ^ ^ and so p n < +oo. It is obvious that p n < p n . If p n < p n , 

there is M £ T n such that sup -E'(u) < p n . The closure M of M in M is still in T n , by 

ueA/ 

continuity of E, sup E(u) < p n holds. By the property (i2) of the cohomological index, 

uGM 

we can find a small open neighborhood A G T n of M in A4 such that sup E(u) < p n . As 

ue^4 

it was proved in the proof of Proposition 3.1 in [10], for every symmetric open subset A 

of Ai, there holds i{A) = sup{i(i^)| K is compact and symmetric with K C ^4}. So we 

can choose a symmetric compact subset K C A with i{K) > n and sup -E(u) < p n . This 

contradicts to the definition of p n . Therefore 

Set C m = {uen\ {0}| E(u) < p m J(u)} and D m = {u G ^| £?(u) < /x m+ i J(u)}. It 

is clear that C m , D m G S(H), i.e., C m and D m are symmetric subsets of % and do not 

contain 0. 

Theorem 3.10 If p m < Pm+i for some m G N, then the cohomological indices satisfy 

i(C m ) = i(D m ) = m. (3.26) 

Proof . Follow the idea of the proof of Theorem 3.2 in |10| . Suppose p m < p m +i- If we 
set A m = {u G E(u) < p m } and B m = {u G M\ E(u) < /x m+ i}, by the definition 
(|3,24p . we have i(A m ) < m. Assume that i(A m ) < m — 1. Then, by the property (i2) of 
the cohomological index, there exists a symmetric neighborhood N of A m in A4 satisfying 
i(N) = i(A m ). By the equivariant deformation theorem (see [8]), there exists 5 > and an 
odd continuous map i : {u G M \ E(u) < p m + 5} -)■ {u G M \ E(u) < p m - 5} U N = N. 
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Hence i(u G M\ E(u) < a m + S) < m - 1. By (|3.24[) . there exists M G T m such that 
sup £(u) < Hm + 5. So M C {u G M\ E(u) < \i m + 5} and thus i(M) < m - 1. This 
contradicts to the fact that M G .F m . Thus we have i(A m ) = m. By 2-homogeneousness 
of the functionals E, J, the map h : C m — > A m with h(u) = J- — u is odd, from the 

v J ( u ) 

monotonicity (il) of the cohomological index, we have i{C m ) < m. But it is clear that 

A m C C m , we have i(C m ) > in, so i(C m ) = m. 

Since A m C B m and i(A m ) = m, we have i(B m ) > m. Assume that i(B m ) > m + 1. 

As in the proof of Lemma 13.91 there exists a symmetric, compact subset K of B m with 

i(K) > m + 1. Since maxE(u) < ^ m +i, this contradicts to definition (|3.24p . So i(B m ) = 

m. Similar to the above arguments, we also have i(D m ) = m. I 

Remark 3.11 If we consider the following eigenvalue problem, 

E'(u) = nj'(u), u G H r , (3.27) 
then all the results in this section still hold, we only need to replace the space H by H r . 

4 Proof of the main theorems 

Replacing (A, Vi,j) with (—A, — Vi, —7) if necessary, we can assume that A > 0. First, we 



consider the case that condition (**) holds and there exists m > 1 such that n m < A < 
n m+i- Set 

C- = {u G H\ E(u) < n m J(u)}, (4.28) 

C+ = {u G H\ E(u) > fi m+1 J(u)}. (4.29) 

It is easy to see that C_, C+ are two symmetric closed cones in Ti and C_ n C+ = {0}. 
By flSUP we have 

i(C7_ \ {0}) = i{C m ) = i(D m ) = i(U \ C+) = m. (4.30) 
Lemma 4.1 There exist r + > and a > suc/i that ^(u) > a /or u G C + and ||u|| = r + . 



Proof . Let e > be small enough, from (Wi) and (W3), we have |W(a;, z)\ < s\z\ 2 +C £ \z 
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By the Sobolev embedding inequality, for u = (u%, u 2 ) G C+, we can get 



£(u)-^-/Wi</(u)-P(u) 



(4.31) 



f(u) = E(u) - AJ(u) - P(u) 

A 

m+ 

> E(")-^E(")-zf^\ui\ 2 dx 
—£ f RN |u 2 | 2 dx - C £ J RN |ui| p dx - C £ J RN |-u 2 | p dx 

> E(u) - j^E(u) - I f RN biWutfdx - | f RN b 2 (x)\u 2 \ 2 dx 
-C £ J RN |iii| p dx - C £ J rN |u 2 | p dx 

^ ( X " S^T " 2max(^, §))£(u) - C £ J RN M P dx - C £ J RN |u 2 |^dx 

> l(l-^r-2max(|,p||uf-C||ur. 

We remind that in the second inequality of (|4.3ip . the condition (B) has been applied. 

Since p > 2, the assertion follows. I 
Since A > fi m , by (Wi) it holds that 

$(u)<0,VueC_. (4.32) 

Set R + = [0, +00). Following the idea of the proof of Theorem 4.1 in [10] , we have 
Lemma 4.2 Let e = (ei,e2) G % \ C_, there exists r_ > r + such that ^(u) < for 
u € C_ + R + e and ||u|| > r_. 

Proof. Define another norm on ?^ by || u| I y := J RAr (|Fi(x)|+| 7 (x)|+l)|«| 2 dx+/ Rjv (|V 2 (x)| + 
| 7 (x)| + l)|u | 2 dx for u = (u, u). Then the same reason as the proof of Theorem 4.1 in [10 1, 
there exists some constant b > such that ||u + te|| < 6||u + ie||y for every u £ C_, i > 
and some b > 0. That is 

J R iv(|V(ti + iei)| 2 + 6i(x)|n + iei| 2 )dx + J RJV (|V(w + ie 2 )| 2 + b 2 {x)\v + te 2 | 2 )dx 

< b 2 f^mix)] + | 7 (x)| + l)|u + t ei | 2 dx + b 2 f RN (\V 2 (x)\ + | 7 (x)| + l)\v + ie 2 | 2 dx. 

(4.33) 

Let {ufc} be a sequence such that ||ufc|| — > +00 and G C_ + R + e. Set v& = 
(uk,Vk) := TnjnT) then, up to a subsequence, {v^} converges to some v = (uq,vq) weakly 
in T~L and — > uq, — > vq a.e. in R^. Note that Lemma 13.41 is also true for functional 
fnA\Vi(x)\ + | 7 (x)| + l)|u| 2 dx + f RN (\V 2 (x)\ + | 7 (x)| + l)M 2 dx, u = (u, u) G ft, it follows 
from g33J that J rN (I^)| + | 7 (x)| + l)|u | 2 dx + j RN (\V 2 (x)\ + | 7 (x)| + l)M 2 dx > 
So |v| /Oona positive measure set ^o- By (W 2 ) we have 

W(x,u k (x)) W(x, ||u fc [|v fc (x)) 2 
hm n 7775 = hm — rpr- v fc (x) = +00, x G U . 

||Ufc|| z fc-S>oo ||Ufe|| z |Vfc(x)| z 
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By (Wi) and Fatou lemma we can get 

J r n W(x,u k (x))dx 



\ u k\ 



+00, as k — > 00. 



By the arbitrariness of the sequence {u/%}, we have 

J k n W(x,u(x))dx 



-00 



u 



as ||u|| —7- +00 and u G C— + R + e. Noting that 

*(u) _ 1 AJ(u) J RJV W(x,u(x))dx 
||u|| 2 2 || u || 2 ll u ll 2 

and by conditions (B) and (V), for u= (u,v) G H 
J(u 



(4.34) 



(4.35) 



u 



C(/ R iv \u\ 2 dx + J RJV H 2 dx) C(/ RJV 6i(x)|u| 2 dx + J Rjv 6 2 (^)b| 2 dx) 

^ — ^ 777^ O, 



u 



u 



(4.36) 



the assertion follows from (j4T34"j) . P~35|) and ([^736]) . I 

Lemma 4.3 ^/ satisfies the Cerami condition, i.e., for any sequence {u k } in % satisfying 
(1 + HufclD^^Ufc) — > and ^(u^) — > c possesses a convergent subsequence. 

Proof . Let {u^} be a sequence in T~L satisfying (1 + ||iifc||)\l/'(ufc) — > and ^(u^) — > c. 
We claim that {u^} is bounded in %. Otherwise, if lluJI — > 00, we consider v k := tt^St. 

II || 

Then, up to subsequence, we get v& — v in H and — > v a.e. in R^. 
If v 7^ in H, since ^'(u^u^. — > 0, that is to say 

||u fc || 2 - AJ'(ufc) • u fc - J RJV V z W(x,u k (x)) ■ u k dx 
= ||u fc || 2 - 2AJ(u fc ) - J RN V z W(x,u k (x)) ■ u k dx 0, 

from (14.361) . we have |^"m^ < C, so by dividing the left hand side of (14.371) with llu/jl 
there holds 



(4.37) 



V z W(x,u k (x)) ■ u k (x) 



dx 



(4.38) 



>R N \\ u k\\ 

for some constant C > 0. On the other hand, Since v(x) ^ in some positive measure 
set O C R^, so v k (x) 7^ for large k, and |ufc(x)| — > +00 as k — > 00, for any fixed x G f2. 
So by (W2), we have 

Hm |v fc (x)| 2 2 ^ ( , X,U 2 fc(x)) = +00, Vx G n. (4.39) 
By Remark (1) before Theorem ll.il we have 

V z W(x, Ufc(x)) • u fc (x) > 2W(x, u fc (x)). 
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So as k — > +00, we have 



V z W{x,u k (x)) ■ u fe (x) f 2 V z W(x,u k (x)) • u fc (x) 

dx = / v fcW ; — r^T9 dx 



ll u fc|| 2 7{v fe (x)^o} I u fc0*0l 2 

/" , . 1 , .. 2 2W(x,u k (x)) . f . .. , . l2 2W(x, u k (x)) 

> J Klf XW}(x)\v k {x)\ , Ufc(x)|2 dx> y n x { ^}(x)|v*(x)| Mg)|2 dx^oo, 

this contradicts to (j4.38j) . There is another explanation about the above estimate. We 
observe that there exists 5 > such that meas{x E fi| |v(x)| > <5} > 0. Otherwise, Vn E N, 
meas{x E 0| |v(x)| > M = 0. Set f2 n = {x E J2| |v(x)| > ^}, then in Q\ Un^L^"' there 
holds v(x) = 0. But ft \ [Jn=i and f2 have the same measure, it is impossible. We may 
assume meas SI < +00, by Egorov's theorem, there exists a positive measure subset of 
{x E fi| |v(x)| > 5} such that uniformly convergent to v, so for k > K with K large, 
there holds |vfc(x)| > 5/2 in Qq. Thus (14.39H holds in $1q. So there holds 

\M X )\ 2 VzW{X ; U it' UkiX) dx> [ |v fc (x)| 2 ^^Mdx^oo. 
{v fc (x)^o} |ufe(x)| 2 Jn |ufc(x)| 2 

If v = in H, inspired by [Til , we choose t k E [0, 1] such that ^(t k u k ) := max vpfiiifc). 

te[o,i] 

For any /3 > and := (4/3) 1//2 v^ — 0, by Lemma 13.41 and the compactness of P' (see 
Lemma 1.22 in [18]) we have that J(v"fe) — > and J RJV W(x, Vfc(x))dx = P(vjfc) — -P(O) = 
v fc ) = (i*(&v fc ) - P'(0),v fe ) + (P'(0),v fc ) -> as k -> 00, here £ fc E (0,1). So 
there holds 

*(ifcu fe ) > *(v fc ) = 2/3 - A J(v fc ) - f W(x, v fc (x))dx > /3, 
when is large enough. By the arbitrariness of /3, it implies that 

lim V(t k u k ) = 00. (4.40) 

fc— >oo 

Since ^(0) = 0, *f>(u k ) — > c, we have t k E (0, 1). By the definition of t k , 

(V'(t k u k ),t k u k ) = 0. (4.41) 

From (g30D, (|QT|) , we have 



*(t fc u fc ) - ^(^'(tfcUfcJ.ifcUfc) 

= V (^^(MfcU^)) • ifcUfc(x) - W(x,t k u k (x))) dx -)■ 00. 



(4.42) 



By (W4), there exists 9 > 1 such that 



Jr^ (l^z^O^ u fc(^)) • u fc(x) - W(x, u fe (x))) dx 

> \ J RN (V z W(x, t k u k (x)) ■ t k \i k {x) - W(x, t k u k (x))) dx, 



(4.43) 
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Hence 




jv V 2 



-V z W(x, Ufc(x)) • u fc (x) - W(x, u fc (x)) dx -> oo. 



(4.44) 



On the other hand 



1 



jv V 2 




(|4.44p and f|4.45|) are contradiction. Hence {u^} is bounded in H. So up to a subsequence, 
we can assume that — u for some %. 



Since ^'(u^) = P/(u.fc) — AJ'(iifc) — P'(iifc) — >• and J', P' are compact, we have that 
E'(u k ) -> AJ'(u) + P'(u) in ft. So 

(P'K), u fc - u) = (P'(ufc) - (AJ'(u) + ^(u)), u fc - u) + (AJ'(u) + P'(u), u fc - u) -> 0. 



Remark 4.4 If we replace the space % by % r , then Lemma l4.lt 14-21 14.31 also hold. 

Proof of Theorem 11.11 Define D_, S+, Q, H as Theorem 12. 1\ then from Lemma 14.11 
'I'(u) > a > for every u 6 S+, from Lemma 14.21 ^(u) < for every u € D- U P and ^ 
is bounded on Q. Applying Lemma 14.31 it follows from Theorem 12.11 that '3/ has a critical 
value d > a > 0. Hence u is a non-trivial weak solution of (jl.ip . 

For the cases < A < n\ or (x) = = V2 (a;), sg t C- = {0} an d C+ = 7i, it is easy 
to see that the arguments above are also valid. The proof of Theorem 11.11 is complete. I 
Proof of Theorem 11.21 By Remarks 13.111 and 14.41 the proof is the same as that of 

Theorem ll.il we only need to replace the space H by % r . I 
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